Commodity recognition apparatus and commodity recognition method

ABSTRACT

In accordance with one embodiment, a commodity recognition apparatus detects, from a captured image, an object imaged in the captured image and extracts an appearance feature amount of the object from the image of the object; compares the extracted appearance feature amount with feature amount data of a dictionary file in which feature amount data indicating the surface information of a commodity is stored for each recognition target commodity to calculate a similarity degree indicating how similar the appearance feature amount is to the feature amount data for each recognition target commodity; recognizes whether or not the object is a commodity based on the calculated similarity degree; and specifies and notifies the reason in a case in which the object is not recognized as a commodity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of application Ser. No. 14/537,963filed Nov. 11, 2014, the entire contents of which are incorporatedherein by reference.

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2013-240021, filed Nov. 20, 2013, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a commodity recognitionapparatus which recognizes a commodity from a captured image and acommodity recognition method for enabling a computer to function as thecommodity recognition apparatus.

BACKGROUND

There is a technology in which an object is recognized according to asimilarity degree calculated by extracting an appearance feature amountof the object from the image data of the object photographed by an imagecapturing section and comparing the extracted appearance feature amountwith the feature amount data of a reference image of each objectpre-registered in a recognition dictionary file. Such a technology iscalled as a general object recognition, and various recognitiontechnologies are disclosed in the following document.

Keiji Yanai “Current status and future direction of general objectrecognition”, Journal of Information Processing Society, Vol. 48, No.SIG16 [Search on Heisei 22 Aug. 10], Internet <URL:http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

In addition, the technology carrying out the general object recognitionby area division of the image for each object is described in thefollowing document.

Jamie Shotton etc, “Semantic Texton Forests for Image Categorization andSegmentation”, [Search on Heisei 22 Aug. 10], Internet<URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.145.3036&rep=repl&type=pdf>

In recent years, for example, the general object recognition technologydescribed above is proposed to be applied to a recognition apparatus forrecognizing a commodity, especially a commodity on which a barcode isnot applied, such as vegetables, fruits and the like purchased by acustomer in a checkout system (POS system) of a retail store. In thiscase, an operator (a shop clerk or a customer) holds the recognitiontarget commodity over the image capturing section. In this way, in thecommodity recognition apparatus, the commodity is recognized from acaptured image and the information (for example, a commodity name)indicating the recognition result is displayed on a display section.

However, the commodity is not always recognized. For example, there is acase in which the commodity is not recognized because the commodity isshield by the hand of the operator. Alternatively, there is a case inwhich the commodity is not recognized because the similar feature amountdata is not registered in the recognition dictionary file. Further,there is a case in which the commodity is not recognized because thecommodity recognition program is frozen.

The commodity recognition apparatus has no reaction in a case in whichthe commodity is not recognized. As a result, it is difficult todetermine the reason why the commodity is not recognized, and mostoperators feel stress because they tried to change the position and thedirection to hold the commodity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of a store checkout system;

FIG. 2 is a block diagram illustrating the hardware constitution of ascanner device and a POS terminal;

FIG. 3 is a schematic view illustrating the structure of dictionary datafor each commodity stored in a recognition dictionary file;

FIG. 4 is a schematic view illustrating a main memory area formed in aRAM of the scanner device;

FIG. 5 is a flowchart illustrating the procedure of a main informationprocessing executed by a CPU of the scanner device according to acommodity recognition program;

FIG. 6 is a schematic view illustrating an example of a screen of atouch panel when a commodity recognition operation is carried out;

FIG. 7 is a schematic view illustrating an example of a screen of thetouch panel when a commodity recognition operation is not carried out;

FIG. 8 is a schematic view illustrating an example of a notificationscreen in a case in which an object is not recognized as a commodity;

FIG. 9 is a schematic view illustrating another example of thenotification screen in a case in which an object is not recognized as acommodity; and

FIG. 10 is a schematic view illustrating another example of thenotification screen in a case in which an object is not recognized as acommodity.

DETAILED DESCRIPTION

In accordance with one embodiment, a commodity recognition apparatuscomprises a recognition module, a specification module and anotification module. The recognition module recognizes, from a capturedimage, a commodity imaged in the captured image. The specificationmodule specifies the reason in a case in which the commodity cannot berecognized by the recognition module. The notification module notifiesan operator of the reason specified by the specification module.

Hereinafter, the embodiment of the commodity recognition apparatus isdescribed with reference to the accompanying drawings. In the presentembodiment, a scanner device 1 constituting a store checkout system of aretail store which deals in vegetables, fruits and the like is providedwith the functions of the commodity recognition apparatus.

FIG. 1 is an external view of the store checkout system. The systemincludes the scanner device 1 serving as a registration section forregistering a commodity to be purchased by a customer and a POS (PointOf Sales) terminal 2 serving as a settlement section for processing thepayment of the customer. The scanner device 1 is arranged on a checkoutcounter 3. The POS terminal 2 is arranged on a register table 4 across adrawer 5. The scanner device 1 and the POS terminal 2 are electricallyconnected with each other through a communication cable 300 (refer toFIG. 2).

The scanner device 1 is provided with a keyboard 11, a touch panel 12and a customer display 13 as the devices required for the reading of thecommodity. These display/operation devices are arranged on a thinrectangular housing 1A constituting a main body of the scanner device 1.

An image capturing section 14 is arranged inside the housing 1A. Arectangular-shaped reading window 1B is formed at the front side of thehousing 1A. The image capturing section 14 includes a CCD (ChargeCoupled Device) image capturing element as an area image sensor and adrive circuit thereof, and an image capturing lens for focusing theimage of an image capturing area on the CCD image capturing element. Theimage capturing area refers to an area of a frame image focused on thearea of the CCD image capturing element through the image capturing lensfrom the reading window 1B. The image capturing section 14 outputs theimage of the image capturing area focused on the CCD image capturingelement through the image capturing lens. The image capturing section 14may be a CMOS (complementary metal oxide semiconductor) image sensor.

The POS terminal 2 is provided with a keyboard 21, an operator display22, a customer display 23 and a receipt printer 24 as the devicesrequired for the settlement. The keyboard 21, the operator display 22and the customer display 23 within these devices are arranged on ahousing constituting a main body of the POS terminal 2. The receiptprinter 24 is arranged inside the housing, and a receipt printed by theprinter is issued from a receipt issuing port formed at the front sideof the housing.

The checkout counter 3 is in an elongated shape along a customer passageat the rear side thereof. The register table 4 is arranged at a sideopposite to the customer passage with respect to the checkout counter 3at a substantially right angle to the checkout counter 3. Specifically,the register table 4 is located at the end of the checkout counter 3 atthe downstream side of a movement direction of a customer moving alongthe checkout counter 3. Therefore, the checkout counter 3 and theregister table 4 are arranged in an L-shape to define a space for a shopclerk in charge of settlement, i.e., so called casher.

At the approximate center of the checkout counter 3, the housing 1A ofthe scanner device 1 is vertically arranged such that the keyboard 11,the touch panel 12 and the reading window 1B are directed to the spacefor a shop clerk (cashier). The display for customer 13 of the scannerdevice 1 is arranged on the housing 1A, facing to the customer passage.

A first upper surface portion of the checkout counter 3 at the upstreamside thereof through the scanner device 1 in the customer movementdirection serves as a space for placing a shopping basket 6 in which anunregistered commodity M purchased by a customer is held. On the otherside, a second upper surface portion at the downstream side through thescanner device 1 serves as another space for placing a shopping basket 7in which a commodity M registered by the scanner device 1 is held.

FIG. 2 is a block diagram illustrating the hardware constitutions of thescanner device 1 and the POS terminal 2. The scanner device 1 comprisesa scanner section 101 and an operation-output section 102. The scannersection 101 includes a CPU (Central Processing Unit) 111, a ROM (ReadOnly Memory) 112, a RAM (Random Access Memory) 113 and a connectioninterface 114 which are connected with each other via a bus line 115including an address bus and a data bus. The scanner section 101 furtherconnects the image capturing section 14 with the bus line 115 through aninput/output circuit (not shown).

The CPU 111 is a central part of a computer. The CPU 111 controls eachsection to achieve various functions of the scanner device 1 accordingto an operating system or an application program.

The ROM 112 is a main storage part of the computer. The ROM 112 storesthe operating system and the application program mentioned above. Asoccasion demands, ROM 112 also stores data required to execute variousprocessing by the CPU 111.

The RAM 113 is also a main storage part of the computer mentioned above.The RAM 113 stores data required to execute various processing by theCPU 111 as needed. Further, the RAM 113 is also used as a work area forthe CPU 111 when various processing, for example, the processing ofstoring the frame image captured by the image capturing section 14, isexecuted.

The operation-output section 102 includes a connection interface 116.The operation-output section 102 connects a bus line 117 with theconnection interface 116, and connects the keyboard 11, the touch panel12 and the display for customer 13 with the bus line 117 through aninput/output circuit (not shown) to realize the functions of theoperation-output section 102. The touch panel 12 includes a panel typedisplay 12 a and a touch panel sensor 12 b overlaid on the screen of thedisplay 12 a. The operation-output section 102 further connects a speechsynthesis section 118 with the bus line 117. The speech synthesissection 118 outputs a speech or voice signal to a speaker 17 in responseto a command input via the bus line 117. The speaker 17 converts thevoice signal into a voice to output it.

The connection interface 114 of the scanner section 101 and theconnection interface 116 of the operation-output section 102 areconnected with each other through the communication cable 300. Throughthe connection, the data signal from the scanner section 101 is sent tothe operation-output section 102, and the operation of theoperation-output section 102 is controlled.

The POS terminal 2 also carries a CPU 201 as a main body of the controlsection. The CPU 201 is connected with a ROM 203, a RAM 204, anauxiliary storage section 205, a communication interface 206 and aconnection interface 207 via a bus line 202. In addition, the keyboard21, display for operator 22, display for customer 23, printer 24 anddrawer 5 are respectively connected with the bus line 202 via aninput-output circuit (not shown).

The communication interface 206 is connected with a store server (notshown) serving as a center of the store via a network such as a LAN(Local Area Network) and the like. Through this connection, the POSterminal 2 can perform a transmission/reception of data with the storeserver.

The connection interface 207 is connected with the two connectioninterfaces 114 and 116 of the scanner apparatus 1 via a communicationcable 300. Through the connection, the POS terminal 2 receivesinformation from the scanner section 101 of the scanner apparatus 1. Onthe other hand, the scanner apparatus 1 accesses the data file stored inthe auxiliary storage section 205 of the POS terminal 2.

The auxiliary storage section 205, which is, for example, a HDD (HardDisk Drive) device or a SSD (Solid State Drive) device, further storesdata files such as a recognition dictionary file 30, a PLU file 40 andthe like, in addition to various programs.

FIG. 3 is a schematic view illustrating the structure of the dictionarydata for each commodity stored in the recognition dictionary file 30. Asshown in FIG. 3, a plurality of feature amount data are stored in therecognition dictionary file 30 for each recognition target commodity inassociation with a commodity name and a commodity ID for identifying thecommodity. The feature amount data is obtained by extracting, from areference image obtained by photographing a commodity identified withthe corresponding commodity ID, an appearance feature amount serving asthe surface information (appearance shape, tint, pattern, andconcave-convex state and the like) of the commodity, and representingthe appearance feature amount in the form of parameters. Feature amountdata 1-N obtained when a commodity is observed from various directionsare respectively stored for the commodity. The number N of featureamount data for one commodity is not fixed. The number N of featureamount data is different from one another according to commodity. Inaddition, the commodity name is not necessarily contained in thedictionary data for each commodity.

The PLU file 40 stores commodity data such as a unit price serving asthe price per unit in association with the commodity ID of eachcommodity. When the commodity ID of a commodity for sale is specified inthe scanner device 1, the CPU 201 reads the unit price associated withthe commodity ID from the PLU file 40 and registers the sales data ofthe commodity for sale with the unit price. Further, it is possible tocollectively manage the dictionary data for each commodity stored in therecognition dictionary file 30 and the data for each commodity stored inthe PLU file 40 in one data file.

In the store checkout system with such a constitution, the imagecapturing section 14 of the scanner device 1 functions as an imagecapturing module for photographing the commodity held over the readingwindow 1B by the operator (shop clerk). The touch panel 12 functions asa display module for displaying the image captured by the imagecapturing module (image capturing section 14). The CPU 111 functions asa detection module, an extraction module, a calculating module, therecognition module, the specification module and the notificationmodule.

The detection module detects, from an image captured by the imagecapturing section 14, an object imaged in the image. The extractionmodule analyzes the image of the object detected from the captured imageand extracts the appearance feature amount of the object. Thecalculating module compares the appearance feature amount of the objectextracted from the object image with the feature amount data of eachrecognition target commodity stored in the recognition dictionary file30 to calculate, for each recognition target commodity, a similaritydegree indicating how similar the appearance feature amount is to thefeature amount data. The recognition module recognizes, from thecaptured image, the commodity imaged in the captured image based on thecalculated similarity degree for each recognition target commodity. Thespecification module specifies the reason in a case in which thecommodity cannot be recognized. The notification module notifies theoperator of the reason specified by the specification module.

These modules are realized by executing processing according to thecommodity recognition program by the CPU 111 of the scanner device 1.The commodity recognition program is stored in, for example, the ROM112. As shown in FIG. 4, the scanner device 1 forms a determinedcommodity memory 41, a confirmation commodity memory 42, a candidatecommodity memory 43, a pattern memory 44 and a color memory 45 in theRAM 113 as the memory areas required to execute the processing.

All the determined commodity memory 41, the confirmation commoditymemory 42 and the candidate commodity memory 43 have an area for storingthe commodity ID and the later described similarity degree as a pair.Particularly, the determined commodity memory 41 and the confirmationcommodity memory 42 store only one pair of commodity ID and similaritydegree. On the other hand, the candidate commodity memory 43 can store aplurality of pairs of commodity ID and similarity degree. Specifically,the determined commodity memory 41 stores the similarity degree and thecommodity ID of a commodity of which the similarity degree is greaterthan a first threshold value A. The confirmation commodity memory 42stores the similarity degree and the commodity ID of a commodity ofwhich the similarity degree is equal to or smaller than the firstthreshold value A and greater than a second threshold value B smallerthan the first threshold value A. The candidate commodity memory 43stores the similarity degree and the commodity ID of a commodity ofwhich the similarity degree is equal to or smaller than the secondthreshold value B and greater than a third threshold value C smallerthan the second threshold value B.

The pattern memory 44 stores pattern data obtained by patterning variousshapes of the hand of the operator of the scanner device 1 when theoperator holds the commodity over the reading window 1B. The hand of theoperator is one reason why the commodity cannot be recognized by therecognition module. That is, there is a case in which the hand of theoperator shields the commodity when the operator holds the commodityover the reading window 1B, and as a result, the commodity cannot berecognized. Thus, the pattern memory 44 is arranged in the presentembodiment to specify such a reason.

The color memory 45 stores color data set for each reason when thecommodity cannot be recognized. The reason why the commodity cannot berecognized further includes a reason that the position of the commodityis too far, or a reason that the position of the commodity is shifted,in addition to the reason that the hand of the operator shields thecommodity. That is, there is a case in which the commodity cannot berecognized because the position of the commodity held over the readingwindow 1B is shifted from the image capturing area. There is a case inwhich the commodity cannot be recognized because the position of thecommodity held over the reading window 1B is too far, and as a result,the resolution of the commodity image is low. Alternatively, there is acase in which the feature amount data of the commodity is not set in therecognize dictionary memory 30. In the present embodiment, thedifferences of these reasons can be recognized according to thedifferences of the colors of frame borders surrounding the recognitiontarget object.

When the commodity recognition program is started, the CPU 111 startsthe procedure shown in the flowchart in FIG. 5. The procedure of theprocessing described below including the procedure shown in FIG. 5 isjust described as an example, and various processing procedures can beused properly to achieve the same result.

First, the CPU 111 outputs an ON-signal of image capturing to the imagecapturing section 14 (ACT 1). The image capturing section 14 starts tophotograph the image capturing area according to the ON-signal of imagecapturing. The frame images of the image capturing area captured by theimage capturing section 14 are stored in the RAM 113 in sequence.

After outputting the ON-signal of image capturing, the CPU 111sequentially acquires the frame images stored in the RAM 113 anddisplays the images on the display 12 a (ACT 2). The CPU 111 detects,from the frame image, the object imaged in the frame image (ACT 3:detection module). For example, the CPU 111 extracts a contour line andthe like from the binary image of the frame image. Then the CPU 111tries to extract the contour of the object imaged in the frame image.After the contour of the object is extracted, the CPU 111 regards thatthe object is detected from the frame image (ACT 4).

In a case in which the object cannot be detected from the frame image(NO in ACT 4), the CPU 111 returns to execute the processing in ACT 2.That is, the CPU 111 acquires a next frame image. Then the CPU 111 triesto detect the object from the frame image.

In a case in which the object is detected from the frame image (YES inACT 4), the CPU 111 specifies, for example, a minimum rectangular-shapedimage area containing the contour of the object as the image area of theobject. Then as shown in FIG. 6 or FIG. 7, the CPU 111 displays a frameborder L1 surrounding the image area of the object (ACT 5). The frameborder L1 is formed in a rectangular shape in FIG. 6 or FIG. 7, however,the shape of the frame border L1 is not limited to this. The frameborder L1 may be formed in a rhombus, circle or ellipse. Further, theframe border L1 is exemplified as a dotted line in FIG. 6 or FIG. 7,however, it is not limited to this, and the frame border L1 may be asolid line, one dotted line and the like.

After displaying the frame border L1, the CPU 111 extracts theappearance feature amount such as the shape, the surface tint, thepattern, the concave-convex state and the like of the object from theimage in the area (ACT 6: extraction module).

After the appearance feature amount is extracted, the CPU 111 accessesthe POS terminal 2 through the connection interface 115. Then the CPU111 sequentially compares the appearance feature amount with the featureamount data of each recognition target commodity stored in therecognition dictionary file 30 to calculate the similarity degree withthe appearance feature amount for each recognition target commodity (ACT7: calculating module).

At this time, in a case in which a similarity degree greater than thefirst threshold value A is calculated, the CPU 111 writes the similaritydegree and the commodity ID of the commodity into the determinedcommodity memory 41. In a case in which the data is already written inthe determined commodity memory 41, the CPU 111 retains the data withthe greater similarity degree in the determined commodity memory 41.

In a case in which a similarity degree equal to or smaller than thefirst threshold value A but greater than the second threshold value B iscalculated, the CPU 111 writes the similarity degree and the commodityID of the commodity into the confirmation commodity memory 42. In thiscase, if the data is already written in the confirmation commoditymemory 42, the CPU 111 retains the data with the greater similaritydegree in the confirmation commodity memory 42.

In a case in which a similarity degree equal to or smaller than thesecond threshold value B but greater than the third threshold value C iscalculated, the CPU 111 writes the similarity degree and the commodityID of the commodity into the candidate commodity memory 43. At thistime, if the data is already written in the candidate commodity memory43, the CPU 111 sorts the data in the candidate commodity memory 43 inthe descending order of similarity degree.

The CPU 111 confirms whether or not the similarity degree greater thanthe third threshold value C is calculated (ACT 8: recognition module).In a case in which the data is not written in the determined commoditymemory 41, the confirmation commodity memory 42 or the candidatecommodity memory 43, the similarity degree greater than the thirdthreshold value C is not calculated. In this case (NO in ACT 8), the CPU111 specifies the reason why the commodity cannot be recognized (ACT 9:specification module).

Specifically, the CPU 111 sequentially compares the image in the areasurrounded by the frame border L1 with each pattern data set in thepattern memory 44 to calculate a correspondence rate. For example, whena pattern data corresponding to the image at a probability as high as80% is detected, the CPU 111 specifies that the reason why the commoditycannot be recognized is the hand of the operator.

On the contrary, for example, in a case in which the pattern datacorresponding to the image at a probability as high as 80% is notdetected, the CPU 111 measures the size of the area surrounded by theframe border L1. In a case in which the size is smaller than a presetminimum value, the CPU 111 specifies that the reason why the commoditycannot be recognized relates to the position of the commodity. In a casein which the size of the area surrounded by the frame border L1 isgreater than the minimum value, the CPU 111 specifies that the reason isthat the data is not set in the recognize dictionary memory 30. In ACT9, the CPU 111 may first determine whether or not the size of the areasurrounded by the frame border L1 is smaller than the minimum value, andthen execute the processing of comparing with each pattern data in thepattern memory 44 to specify the reason if the size is smaller than theminimum value.

After the reason why the commodity cannot be recognized is specified,the CPU 111 detects the color data set for the reason by reference tothe color memory 45. Then the CPU 111 changes the color of the frameborder L1 to the color of the color data (ACT 10: notification module).

Sequentially, the CPU 111 returns to ACT 2 to acquire a next frameimage. Then the CPU 111 executes the processing following ACT 3 again.

In a case in which a similarity degree greater than the third thresholdvalue C is calculated (YES in ACT 8), the CPU 111 stops the notificationprocessing carried out in ACT 10 (ACT 11). Sequentially, the CPU 111executes a determination processing (ACT 12). The content of thedetermination processing varies in a case in which the data is writtenin the determined commodity memory 41, in a case in which the data iswritten in the confirmation commodity memory 42, and in a case in whichthe data is written in the candidate commodity memory 43.

In a case in which the data is written in the determined commoditymemory 41, the CPU 111 determines the commodity specified with thecommodity ID stored in the determined commodity memory 41 as thecommodity held over the reading window 1B. Then the CPU 111 generates avoice from the speaker 17 to notify that the commodity is determined.

In a case in which the data is written in the confirmation commoditymemory 42, the CPU 111 displays, on the display 12 a, an image of a namebutton which displays the name of the commodity specified with thecommodity ID stored in the confirmation commodity memory 42 in a buttonframe. Then the CPU 111 waits until the name button is touched. When itis detected that the name button is touched according to the signaldetected by the touch panel sensor 12 b, the CPU 111 determines thecommodity specified with the commodity ID stored in the confirmationcommodity memory 42 as the commodity held over the reading window 1B.Then the CPU 111 generates a voice from the speaker 17 to notify thatthe commodity is determined. On the contrary, in a case in which it isdetected that a screen area other than the name button is touched, theCPU 111 deletes the image of the name button. At this time, thecommodity is not determined.

In a case in which the data is written in the candidate commodity memory43, the CPU 111 displays, on the display 12 a, the images of namebuttons which display the names of the commodities specified with thecommodity IDs stored in candidate commodity memory 43 in button frames.Then the CPU 111 waits until any of the name buttons is touched. When itis detected that one name button is touched according to the signaldetected by the touch panel sensor 12 b, the CPU 111 determines thecommodity corresponding to the name displayed in the name button as thecommodity held over the reading window 1B. Then the CPU 111 generates avoice from the speaker 17 to notify that the commodity is determined. Onthe contrary, in a case in which it is detected that an area other thanthe name button is touched, the CPU 111 deletes the image of the namebutton. At this time, the commodity is not determined.

The CPU 111 determines whether or not the commodity is determined (ACT13). In a case in which the commodity is not determined (NO in ACT 13),the CPU 111 returns to ACT 2 to acquire a next frame image. Then the CPU111 executes the processing following ACT 3 again.

In a case in which the commodity is determined (YES in ACT 13), the CPU111 outputs the commodity ID of the determined commodity to the POSterminal 2 through the connection interface 115 (ACT 14).

Now assume that the shop clerk holds an object M1 over the readingwindow 1B as shown in FIG. 6. In this case, almost the entire object M1is photographed by the image capturing section 14. In the scanner device1, the rectangular-shaped frame border L1 indicated by a dotted line isdisplayed on the touch panel 12 in a manner of surrounding therecognition area of the object. The appearance feature amount of theobject M1 is extracted and compared with the feature amount data of eachcommodity stored in the recognition dictionary file 30.

Herein, it is assumed that the feature amount data of which thesimilarity degree with the appearance feature amount of the object M1 isgreater than the first threshold value A is set in the recognitiondictionary file 30 as the feature amount data of a commodity “banana”.In this case, it is determined in the scanner device 1 that the objectM1 held over the reading window 1B is the commodity “banana”, and thecommodity ID of the commodity “banana” is output to the POS terminal 2.As a result, the sales data of the commodity “banana” is registered inthe POS terminal 2.

In a case in which the similarity degree between the feature amount dataof the commodity “banana” and the appearance feature amount of theobject M1 is greater than the second threshold value B but smaller thanthe first threshold value A, as shown in FIG. 6, a button image N1 ofthe commodity “banana” is displayed on the touch panel 12 in the scannerdevice 1. If the shop clerk touches the button image N1, the commodityID of the commodity “banana” is output to the POS terminal 2. As aresult, the sales data of the commodity “banana” is registered in thePOS terminal 2.

On the other hand, assume that the shop clerk holds an object M2 overthe reading window 1B in a state in which the object M2 is almostentirely shield by the hand of the shop clerk, as shown in FIG. 7. Inthis case, the entire back of the hand including the object M2 isrecognized as one object in the scanner device 1. Then in the scannerdevice 1, the rectangular-shaped frame border L1 indicated by a dottedline is displayed on the touch panel 12 in a manner of surrounding therecognition area of the object. The appearance feature amount of theobject is extracted and compared with the feature amount data of eachcommodity stored in the recognition dictionary file 30.

At this time, the feature amount data of which the similarity degreewith the appearance feature amount of the object is greater than thethird threshold value C is not set in the recognition dictionary file 30in most cases. In this case, conventionally, there is no change in thescreen of the touch panel 12. There is no voice generated from thespeaker, either. That is, the commodity recognition apparatus has noreaction. Thus, the shop clerk cannot determine whether the commodity isbeing recognized or the commodity cannot be recognized, which makes theshop clerk feel stress.

On the contrary, as shown in FIG. 8, the line type of the frame borderL1 is changed from the dotted line to a solid line in the presentembodiment. The color of the frame border L1 is also changed to a colorcorresponding to the reason why the commodity cannot be recognized. Forexample, it is assumed in the scanner device 1 that the image in thearea surrounded by the frame border L1 is sequentially compared witheach pattern data set in the pattern memory 44, and as a result, apattern data corresponding to the image at a high probability isdetected. In this case, it is specified in the scanner device 1 that thehand of the shop clerk is the reason why the commodity cannot berecognized. Then the color of the frame border L1 is changed to a color(for example, red) corresponding to the reason.

In this way, as the color of the frame border L1 is changed to red, theshop clerk is aware that the commodity N1 cannot be recognized becausehis/her hand shields the commodity N1. At this time, the shop clerkholds the commodity N1 over the reading window 1B in a manner that thecommodity N1 is not shield by the hand, thus, the shop clerk does notfeel stress.

On the other hand, though the commodity is held over the reading window1B, there is a case in which the commodity cannot be recognized becausethe position of the commodity is too far from the reading window 1B, andas a result, the resolution of the commodity image is low. In this case,as the size of the commodity image area is smaller than the presetminimum value, thus, it is specified in the scanner device 1 that thereason why the commodity cannot be recognized is the position of thecommodity. Then the color of the frame border L1 is changed to a color(for example, green) corresponding to the reason. In this way, as thecolor of the frame border L1 is changed to green, the shop clerk isaware that the commodity cannot be recognized because the commodity istoo far from the reading window 1B. At this time, the shop clerk onlyneeds to move the commodity closer to the reading window 1B, thus, theshop clerk does not feel stress.

Further, there is a case in which the commodity cannot be recognizedbecause the feature amount data of the commodity is not registered inthe recognition dictionary file 30. In this case, the color of the frameborder L1 is changed to, for example, yellow. In this way, as the colorof the frame border L1 is changed to yellow, the shop clerk is awarethat the commodity cannot be recognized because the feature amount dataof the commodity is not registered in the recognition dictionary file30. In this case, for example, the shop clerk only needs to operate thekeyboard 11 of the scanner device 1 to register the commodity throughmanual input, thus, the shop clerk does not feel stress.

As stated above, in accordance with the present embodiment, therectangular-shaped frame border L1 is displayed on the touch panel 12 ina manner of surrounding the recognition area of the object. When theobject cannot be recognized, the color of the frame border L1 ischanged. As a result, as the color of the frame border L1 is changed,thus, the operator can easily be aware why the commodity cannot berecognized. There is another advantage that the operator can confirmthat the program is not frozen.

The present invention is not limited to the embodiment described above.

For example, it is exemplified in the embodiment described above thatnot only is the color of the frame border L1 changed, but also the linetype of the frame border L1 is changed from the dotted line to the solidline in a case in which the object cannot be recognized, however, theline type may be remained to be the dotted line or the solid linewithout being changed.

Further, the reason why the object cannot be recognized may be notifiedin a display form other than the frame border. For example, as shown inFIG. 9, the operator may be notified of the reason by displaying an iconC1 on the display 12. Incidentally, FIG. 9 shows an example of an iconwhich is displayed when the commodity cannot be recognized because thehand of the operator shields the commodity. For example, in a case inwhich the commodity cannot be recognized because the commodity is toofar from the reading window B, an icon of a mark “?” is displayed. Inthis way, the operator can easily be aware of the reason why the objectcannot be recognized.

As shown in FIG. 10, the operator may be notified of the reason bydisplaying a message D1 indicating the reason. In this case, the messageD1 may be given through a voice under the action of the speech synthesissection 118. Further, the message D1 may be given by both voice anddisplay.

For example, it is also applicable to display the frame bordersurrounding the image area of the object or change the color of theframe border when the reason is the position of the object, and todisplay an icon when the reason is the shielding by the hand, and viceversa.

The present invention is not limited to the commodity recognitionapparatus in which the general object recognition technology is applied.For example, the present invention may also be applied to a commodityrecognition apparatus which detects a barcode image from an imagecaptured by an image capturing module and decodes the barcode of thebarcode image to recognize a commodity.

Further, the method of recognizing the hand of the operator is notlimited to the pattern recognition. For example, it may be recognizedthat the reason is the hand of the operator in a case in which the ratioof a flesh color area to the image area is greater than a given ratio.

In addition, the transfer of the commodity recognition apparatus isgenerally carried out in a state in which the programs such as commodityrecognition program are stored in the ROM. However, the preset inventionis not limited to this. The commodity recognition program and the liketransferred separately from a computer device may be written in awritable storage device of the computer device through an operation of auser and the like. The transfer of the commodity recognition program andthe like may be carried out by recording the program in a removablerecording medium, or through a communication via a network. The form ofthe recording medium is not limited as long as the recording medium canstore programs like a CD-ROM, a memory card and the like, and isreadable by an apparatus. Further, the function realized by an installedor downloaded program can also be realized through the cooperation withan OS (Operating System) installed in the apparatus.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the invention. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinvention. The accompanying claims and their equivalents are intended tocover such forms or modifications as would fall within the scope andspirit of the invention.

1. (canceled)
 2. A commodity recognition apparatus comprising: an imagesensor which has an area image sensor to photograph a commodity andgenerate a captured image of the commodity; a display configured todisplay the image captured by the image sensor; a memory configured tostore computer-readable instructions; and a central processing unitwhich processes the captured image according to the computer-readableinstructions causing the commodity recognition apparatus to: recognizethe commodity imaged in the captured image; specify a reason why thecommodity cannot be recognized in a case in which the commodity cannotbe recognized; and notify a specified reason by a category of a framedisplayed in the display.
 3. The commodity recognition apparatusaccording to claim 2, wherein the specified reason is notified bydifference of colors of the frame.
 4. The commodity recognitionapparatus according to claim 2, wherein the specified reason is notifiedby the frame or an icon.
 5. The commodity recognition apparatusaccording to claim 2, wherein the central processing unit detects, fromthe image captured by the image sensor, an object imaged in the image,and the frame is displayed in a manner of surrounding the object.
 6. Thecommodity recognition apparatus according to claim 5, wherein thecentral processing unit measures a size of the frame in a case in whichthe commodity cannot be recognized, and specifies that the reason whythe commodity cannot be recognized is a position of the commodity in acase in which the size is smaller than a preset value.
 7. The commodityrecognition apparatus according to claim 2, further comprising: apattern memory storing pattern data obtained by patterning shapes of ahand of an operator.
 8. The commodity recognition apparatus according toclaim 7, wherein the central processing unit detects, from the imagecaptured by the image sensor, an object imaged in the image, anddetermines whether the reason why the commodity cannot be recognized isthe hand of the operator by reference to the detected object and thepattern data stored in the pattern memory in a case in which thecommodity cannot be recognized.
 9. A commodity recognition methodcomprising: generating a captured image of a commodity with an imagesensor; displaying the image captured by the image sensor in a display;recognizing the commodity imaged in the captured image; specifying areason why the commodity cannot be recognized in a case in which thecommodity cannot be recognized; and notifying a specified reason by acategory of a frame displayed in the display.
 10. The commodityrecognition method according to claim 9, wherein notifying the specifiedreason is carried out by a difference of colors of the frame.
 11. Thecommodity recognition method according to claim 9, wherein notifying thespecified reason is carried out by the frame or an icon.
 12. Thecommodity recognition method according to claim 9, further comprising:detecting, from the image captured by the image sensor, an object imagedin the image; and displaying the frame in a manner of surrounding theobject.
 13. The commodity recognition method according to claim 12,further comprising: measuring a size of the frame in a case in which thecommodity cannot be recognized; and specifying that the reason why thecommodity cannot be recognized is a position of the commodity in a casein which the size is smaller than a preset value.
 14. The commodityrecognition method according to claim 9, further comprising: storingpattern data obtained by patterning shapes of a hand of an operator in apattern memory; detecting, from the image captured by the image sensor,an object imaged in the image; and determining whether the reason whythe commodity cannot be recognized is the hand of the operator byreference to the detected object and the pattern data stored in thepattern memory in a case in which the commodity cannot be recognized.15. A settlement system comprising: the commodity recognition apparatusof claim 2; and a POS terminal for registering sales data of thecommodity recognized by the commodity recognition apparatus.
 16. Thesettlement system of claim 15, wherein the central processing unit ofthe commodity recognition apparatus outputs a commodity ID of thecommodity to the POS terminal in a case in which the commodity can berecognized from the image captured by the image sensor.
 17. Thesettlement system of claim 15, wherein the commodity recognitionapparatus notifies of the specified reason by difference of colors ofthe frame.
 18. The settlement system of claim 15, wherein the commodityrecognition apparatus notifies of the specified reason by the frame oran icon.
 19. The settlement system of claim 15, wherein the centralprocessing unit detects, from the image captured by the image sensor, anobject imaged in the image, and the frame is displayed in a manner ofsurrounding the object.
 20. The settlement system of claim 19, whereinthe central processing unit measures a size of the frame in a case inwhich the commodity cannot be recognized, and specifies that the reasonwhy the commodity cannot be recognized is a position of the commodity ifthe size is smaller than a preset value.
 21. The settlement system ofclaim 15, wherein the commodity recognition apparatus further comprisesa pattern memory storing pattern data obtained by patterning shapes of ahand of an operator, and the central processing unit detects, from theimage captured by the image sensor, an object imaged in the image, anddetermines whether the reason why the commodity cannot be recognized isthe hand of the operator by reference to the detected object and thepattern data stored in the pattern memory in a case where the commoditycannot be recognized.